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SECTION  1.  INTRODUCTION 


The  maintenance  of  avionics  systems  on  current  Air  Force  aircraft  is  plagued  by 
problems  in  the  areas  of  fault  detection  and  fault  isolation.  Despite  the  built-in  test  (BIT) 
capabilities  included  in  many  line-replaceable  units  (LRUs)  and  the  development  of  diagnostic 
aids  now  used  by  flightline  technicians,  there  remains  the  costly  problem  of  unnecessary 
removal,  switchout,  "swaptronics,"  and  servicing  of  LRUs  due  to  missed  diagnoses.  The 
Technology  for  Readiness  and  Sustainment  program,  DO  2  on  Advanced  Diagnostics,  seeks  to 
identify  Advanced  Diagnostic  (AD)  technologies  that  will  provide  improvements  over  current 
BIT  and  flightline  diagnostic  methods  to  provide  more  accurate  fault  isolation.  An  effective 
solution  will  both  reduce  maintenance  expenses  and  improve  the  operational  readiness  of  USAF 
aircraft. 

Two  main  tasks  were  executed  as  a  part  of  this  program.  The  first  task  consisted  of  a 
review  of  current  diagnostic  practices,  issues,  technologies,  and  maintenance  statistics. 

Interviews  of  flightline  maintenance  technicians  and  depot  logistics  personnel  were  conducted  at 
five  different  USAF  facilities.  The  Reliability  Maintainability  Information  System  (REMIS)  was 
accessed  to  collect  data  to  characterize  the  current  scope  of  diagnostic  discrepancies  (NDFs, 
CNDs,  RTOKs,  BCSs,  etc.).  A  large  body  of  reports  was  reviewed  to  obtain  a  more  detailed 
technical  perspective  on  diagnostic  issues. 

The  second  task  was  to  identify  and  assess  candidate  AD  technologies.  Over  20 
technologies  were  identified.  Literature  on  each  technology  was  obtained,  and  a  questionnaire 
was  submitted  to  each  technology  source  requesting  a  standardized  body  of  information  for  each 
AD  concept.  The  Kepner-Tregoe  (K-T)  analysis  method  was  used  to  assess  the  candidate 
technologies  with  respect  to  customer  requirements.  The  K-T  analysis  results  were  combined 
with  qualitative  analysis  to  establish  recommendations  for  future  Air  Force  research  to  mature 
and  verify  promising  AD  technologies. 

Section  2  provides  a  review  of  the  current  diagnostics  environment,  in  terms  of  lessons 
learned  through  interviews  with  maintenance  personnel,  and  data  from  a  maintenance 
information  database  that  reflects  the  extent  of  problems  with  current  diagnostics.  Section  3 
describes  the  method  used  to  identify  and  screen  AD  technology  concepts,  and  documents  the 
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K-T  analysis  method  used  to  assess  the  technologies  against  customer  requirements.  In  Section 
4,  general  results  of  the  technology  analysis  are  presented.  Conclusions  and  recommendations 
are  included  in  Section  5. 


SECTION  2.  REVIEW  OF  CURRENT  DIAGNOSTICS 

2.1.  INTERVIEWS 


2.1.1.  Focus  and  Schedule 


Interviews  were  conducted  with  personnel  involved  in  activities  related  to  diagnostics 
and  maintenance  of  aircraft  avionics  systems  at  several  USAF  facilities.  The  purpose  of  the 
interviews  was  to  hear  the  perspective  of  people  involved  in  maintenance  and  diagnostics  on  a 
daily  basis  regarding  current  maintenance  and  diagnostic  processes,  and  their  perceptions  of 
where  problems  are  occurring  in  the  current  operating  environment.  This  perspective  was  sought 
to  enhance  the  ability  of  the  research  team  to  judge  whether  candidate  AD  technologies  would  be 
feasible  and  effective  if  implemented  for  current-generation  aircraft. 

A  primary  objective  of  this  study  was  to  identify  AD  technologies  that  are  applicable  to 
the  widest  possible  range  of  USAF  Mission/Design/Series  (MDS)  aircraft.  However,  because  of 
the  large  number  of  MDSs  and  the  large  number  of  USAF  facilities  that  operate  and  maintain 
these  aircraft,  it  was  necessary  to  restrict  the  interviews  to  a  small  number  of  representative 
groups  of  AF  maintenance  personnel,  particularly  those  supporting  fighter  aircraft.  Fighters 
present  the  greatest  challenges  to  the  packaging  of  on-board  diagnostic  systems  (space 
limitations,  weight  limitations,  access  limitations,  on-board  resource  limitations),  and  rapid 
maintenance  and  high  reliability  under  combat  conditions  are  vital  to  fighter  operations. 
Personnel  supporting  flightline  maintenance  were  interviewed,  because  first-level  maintenance 
has  the  most  time-critical  diagnostic  and  repair  responsibility.  Depot  personnel  were  consulted, 
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including  both  those  involved  in  “production”  (LRU/SRU  servicing)  and  those  involved  in  the 
development  of  avionics  systems  and  the  associated  diagnostic  and  maintenance  practices. 


2.1.2.  Summary  of  Lessons  Learned 


Key  lessons  learned  during  the  interviews  of  maintenance  and  depot  personnel  are 
summarized  below. 


Positive  Findings 

•  Many  diagnostics  tools  and  processes  work  satisfactorily  (e.g.  EDNA,  DTM  /  CFRS,  many 
BITs,  some  TOs). 

•  There  is  a  high  level  of  can-do  attitude  on  the  part  of  maintenance  personnel. 


Technician  Training 

The  current  level  of  training  for  O-level  and  I-level  technicians  is  perceived  as  inadequate 
to  support  work  requirements.  In  some  organizations,  technicians  are  expected  to  be  qualified  to 
service  multiple  aircraft  types  and  multiple  avionics  systems  on  each  type  of  aircraft;  to  use 
multiple  diagnostic  tools;  and  to  become  proficient  on  newly-introduced  equipment  with  little  or 
no  training. 


FI  Documentation.  TOs 

•  The  TO  update  process  is  slow.  Publication  and  distribution  of  TO  updates  can  take  up  to 
six  months  to  rectify  after  error  corrections  and  FI  improvements  are  identified.  This 
reduces  the  technician’s  confidence  in  the  accuracy  of  the  TOs. 

•  Some  FI  trees  resort  to  “swaptronics”  (trial-and-error  switching  of  two  or  more  LRUs)  at 
certain  stages  to  resolve  ambiguity  groups. 
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Technicians  desire  more  detail  on  electrical  schematics. 


Built  In  Tests  (BIT) 

•  Much  of  the  existing  BIT  works  well. 

•  “Maximum  allowable  execution  times”  for  executing  BITs  on  certain  LRUs  may 
compromise  effectiveness.  For  example,  the  BIT  on  the  APG-70  radar  system  is  restricted 
to  3  minutes,  however,  a  comprehensive  BIT  on  the  radar  system  actually  requires  5  to  30 
minutes  to  complete. 


Possible  Sources  of  CNDs  and  RTOKs 

•  Some  RTOKs  can  be  attributed  to  vertical  compatibility  problems  in  2-  or  3-  level  testing. 
For  example,  the  depot-level  tests  for  an  LRU  may  have  less  stringent  performance 
tolerances  than  O-level  or  I-level  tests. 

•  Some  CNDs  are  believed  to  result  from  intermittent  fault  indications.  These  can  have  a 
number  of  sources,  including: 

♦  Stresses  to  the  system  related  to  operating  conditions  and  environmental  factors 
(temperature,  moisture,  acceleration,  air  pressure,  vibration)  can  stimulate  intermittent 
circuit  interruptions  in  flight.  These  stresses  are  generally  not  duplicated  in  ground-based 
testing. 

♦  LRU  faults  may  sometimes  be  due  to  software  code  execution  errors  associated  with  the 
input  values  that  accompany  particular  operating  conditions.  A  failure  to  duplicate  these 
input  values  and  operating  conditions  during  ground  testing  results  in  CND,  NFFs,  etc. 
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Fault  Information  Handling  in  Multi-Level  Maintenance 


•  Often,  little  or  no  fault  information  (operational  conditions,  symptoms,  fault  indications) 
accompanies  LRUs  to  the  depot  and/or  backshop.  This  is  judged  by  the  depot  and/or 
backshop  technicians  to  hamper  the  diagnostic  process. 

Problems  with  the  CAMS  Maintenance  Data  Collection  System 

•  The  CAMS  system  is  not  integrated  across  bases  and  depots,  preventing  access  to  data  that 
could  be  useful. 

•  The  user  interface  is  very  cumbersome,  complicated,  slow,  and  user-unfriendly.  This  leads 
to  incorrect  data  entry,  sometimes  inadvertently,  sometimes  intentionally.  Technicians  too 
often  spend  more  time  entering  data  than  performing  diagnostics,  repairs,  and  maintenance. 

•  Technicians  receive  insufficient  CAMS  training.  There  is  also  inadequate  linking  between 
using  organizations.  An  improved  CAMS-like  system  (better  user  interface,  linking 
between  all  organizations  and  depots)  would  improve  diagnostics  within  the  current  climate 
by  providing  technicians  with  accurate  maintenance  data  that  encompassed  the  entire  fleet. 

•  Inadequate  tools  for  identification  of  bad-actor  LRUs  and  SRUs. 

•  Inadequate  tools  for  passing  comprehensive  fault  report  data  between  maintenance  levels  to 
aid  diagnosis. 


Bad  Actors 


•  One  source  of  “bad  actors”  (LRUs  that  repeatedly  give  fault  indications  in-flight  but  test  out 
okay  on  the  ground)  is  believed  to  be  the  failure  to  duplicate  the  adverse  conditions  present 
during  flight  (acceleration,  vibration,  temperature,  moisture,  air  pressure,  etc.)  at  the  bench. 
There  is  evidence  from  depot  work  that  so-called  “shake  and  bake”  testing  (testing  while 
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including  some  of  these  influences)  helps  identify  some  bad  actors;  however  this  type  of 
testing  is  too  costly  to  use  for  all  depot-level  testing. 


2.2.  USAF  FLEET  SIZE 


The  size  of  the  USAF  fleet  of  Legacy  aircraft  was  established  as  part  of  an  effort  to 
determine  the  potential  benefit  of  an  AD  system.  Consideration  was  limited  to  active  aircraft  in 
the  current  USAF  inventory  which  have  been  deployed  in  significant  numbers,  and  which  are 
beyond  the  initial  developmental  stage  with  respect  to  maintenance  practices.  The  aircraft 
Mission/Design/Series  (MDS)  included  in  this  top-level  analysis  are  identified  in  Table  2.2, 
along  with  the  total  number  of  units  in  the  active  fleet.  These  data  were  obtained  through  a 
REMIS  query. 


Table  2.1.  Current  USAF  Aircraft  Quantities  by  MDS 


Mission/Design 

Units  | 

A10 

473 

B1 

93 

B52 

95 

C130 

713 

C5 

126 

C141 

175 

F15 

729 

F16 

1985 

TOTAL 

4389 

6 


2.3. 


CHARACTERISTIC  MAINTENANCE  STATISTICS  FOR  PROBLEMATIC 
DIAGNOSTICS 


REMIS  queries  were  run  to  obtain  characteristic  maintenance  statistics  that  reflect  the 
scope  of  diagnostic  problems  in  the  maintenance  of  USAF  Legacy  aircraft.  Statistics  were 
generated  for  a  one-year  period  from  June  1,  1998  to  May  31,  1999.  The  following  general 
avionics  systems  were  used  as  the  focus  for  the  study: 

•  RADAR 

•  Bombing/Navigation 

•  Automatic  Flight  Controls/Instruments 


Electronic  Warfare  avionics  were  excluded  because  many  of  these  components  are 
external  and  do  not  interface  on  any  bus  with  other  systems.  Engine  instrumentation  was 
excluded  because  of  the  high  level  of  attention  already  given  to  engine  health  monitoring  and 
prognostics. 

REMIS  queries  were  run  for  the  various  MDS  aircraft,  including  all  appropriate  Work 
Unit  Codes  (WUCs)  necessary  to  embrace  the  avionics  systems  named  above.  The  list  of  WUCs 
varied  from  MDS  to  MDS,  depending  on  the  avionics  features  of  the  respective  MDS.  The 
information  was  retrieved  from  REMIS  for  the  following  data  parameters: 

TOTAL_FAIL 

NO_DEFECT 

NO_DUPLICATE 
NUM_BENCHCHK 


Counter  of  all  O-level  maintenance  events  in  response  to  fault  report  for 
specified  WUC. 

O-level  maintenance  actions,  in  response  to  TOTAL_FAIL  events,  for 
which  no  defect  was  identified. 

O-level  Could  Not  Duplicate  (CND)  on  avionics  equipment  fault. 

I-level  Bench  Check  Serviceable  (BCS). 
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DEPOT_EVENT  LRU  and  SRU  depot  maintenance  actions  on  nominally  faulty  LRUs  and 
SRUs  received  from  flight  organizations. 

RETEST_OKAY  RTOK,  no  fault  found  during  DEPOT_E VENT . 

REMIS  data  summaries  (summed  across  multiple  WUCs)  for  one  12-month  period  are 
presented  in  Table  2.2.  Several  rates  are  computed  and  included  in  the  table,  including: 


1.  NDFs  per  AC  per  year 

2.  CNDs  per  AC  per  year 

3 .  Depot  Events  per  AC  per  year 

4.  RTOKs  per  AC  per  year 

5.  RTOK  Rate  (RTOKs  per  Depot  Event) 


As  indicated  in  Table  2.2,  the  average  number  of  NDFs  per  AC  per  year  was  47,  and  the 
average  number  of  CNDs  per  AC  per  year  was  9.  These  numbers  illustrate  the  need  for 
improved  diagnostics.  The  average  number  of  Depot  Events  per  AC  per  year  was  4.3,  with  a 
much  higher  rate  of  8.2  for  F-16s.  The  overall  average  RTOK  rate  was  only  about  8%;  however, 
this  number  may  not  reflect  the  true  extent  of  the  RTOK  problem.  Avionics  production 
personnel  at  one  ALC  stated  that  many  Depot  Events  that  are,  in  effect,  RTOKs  are  not  recorded 
as  such  because  some  electronic  adjustment  or  case  repair  is  performed  on  LRUs. 
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Table  2.2.  REMIS  Maintenance  Statistics  for  the  Period  1  June  1998  to  31  May  1999 


Mission/Design 

w 

c 

D 

O 

< 

cu 

o 

h- 

TOTAL_FAIL 

uT 

Q 

s 

H 

O 

LU 

LL 

LU 

D 

o1 

Z 

NO_DUPLlCATE 

(CND) 

X 

o 

X 

o 

z 

LU 

*1- 
■S  C/D 

X  ° 

i  a 

H 

Z 

LU 

> 

o 

Q_ 

LU 

Q 

RETEST_  OKAY 
(RTOK) 

NDF  per  AC  per  Year 

CNDs  per  AC  per  Year 

Depot  Events  per  AC 
per  Year 

RTOKs  per  AC  per 

Year 

RTOK  Rate  (RTOKs 
per  Depot  Event) 

A10 

473 

26.946 

15,020 

2,422 

992 

800 

96 

32 

5.1 

1.7 

0.20 

12.1% 

B1 

93 

19,609 

13,049 

1,678 

654 

145 

16 

140 

18.0 

1.6 

0.17 

11.0% 

B52 

95 

11,705 

6.178 

671 

27 

263 

34 

65 

2.8 

0.36 

13.0% 

Cl  30 

713 

50,764 

23.857 

3,818 

1,193 

185 

36 

33 

MM 

0.3 

0.05 

21.6% 

C5 

126 

25,251 

12,919 

416 

1,770 

154 

5 

103 

msm 

0.04 

C141 

175 

18,895 

6,01  r 

314 

1,316 

590 

14 

34 

1.8 

MM 

0.08 

2.4% 

F15 

729 

112,610 

70,734 

19,711 

6,602 

489 

67 

97 

27.0 

0.7 

0.09 

13.9% 

F-16 

1.985 

94.154 

59.330 

10.578 

3.977 

16,301 

1,231 

30 

■*1 

0.62 

9.0% 

TOTAL 

4,389 

359,934 

207.104 

39.608 

16,531 

18,927 

1.499 

I  47 

1  9.0 

0.34 

SECTION  3.  ANALYSIS  OF  TECHNOLOGIES 


A  survey  of  existing  and  developing  technologies  was  conducted  to  identify  those  with 
the  potential  to  help  solve  the  diagnostics  issue.  The  Kepner-Tregoe  method  was  used  to 
evaluate  candidate  AD  technology  concepts.  The  K-T  analysis  method  is  described  in  this 
section.  The  method  used  to  identify  and  down-select  AD  technology  concepts  for  analysis  is 
also  documented  and  a  questionnaire  sent  to  AD  technology  sources  is  described.  The  analysis 
results  are  discussed  in  Section  4. 


3.1.  DESCRIPTION  OF  KEPNER-TREGOE  METHOD 

The  K-T  method  was  used  to  rank  candidate  AD  technologies  with  respect  to  relative 
merit.  The  specific  implementation  of  the  K-T  method  was  based  on  the  implementation 
developed  in  a  previous  study. 
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3.1.1.  Customer  Requirements 


Based  on  discussions  with  the  USAF  Technical  Monitor,  a  list  of  customer  requirements 
was  generated,  and  the  relative  priority  of  each  requirement  was  assigned  on  a  scale  of  1  to  5. 
The  Customer  requirements  and  priority  rankings  are  listed  in  Table  3.1. 


Table  3.1.  Customer  Requirements  and  Relative  Priority 


No. 

Customer  Requirement 

Priority  (1-5) 

1 

Applicable  to  multiple  Legacy  systems 

5 

2 

Improve  fault  isolation  performance 

5 

3 

Reduce  CND/BCS/UR/RTOK 

5 

D 

Minimal  impact  on  existing  system  design 

5 

5 

Reduce  logistics  footprint 

4 

6 

AEF  deployable 

4 

7 

Minimal  training  requirement 

3 

8 

Deployable  by  2005 

3 

9 

Commonality  of  parts  and  equipment 

3 

10 

Affordable 

2 

11 

Low  risk 

1 

3.1.2.  Evaluation  Criteria 


Evaluation  criteria  were  identified  for  use  to  grade  candidate  AD  technologies.  The 
Evaluation  criteria  differed  from  the  customer  requirements  because  they  were  selected  to 
provide  criteria  that  could  be  more  easily  assessed  using  information  gathered  during  the  study. 
Evaluation  criteria  were  identified  in  the  four  categories  of  Cost,  Schedule,  Technology  Impact, 
and  Risk,  as  shown  in  Table  3.2.  The  Evaluation  criteria  were  mapped  to  the  customer 
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requirements  by  the  application  of  a  Quality  Function  Deployment  (QFD)  Correlation  Matrix, 
which  will  be  discussed  shortly. 


Table  3.2.  Evaluation  Criteria  and  Relative  Importance 


No. 

Evaluation  Criteria 

Relative 

Importance 

Cost 

R&D  Required  to  Mature 

7 

2 

Initial  Acquisition 

6 

3 

Sustaining  Cost 

2 

H 

Benefit 

8 

5 

Schedule  -  Time  Required  to  Mature 

6 

Technology  Impact 

6 

Aircraft  Availability 

10 

7 

Aircraft  Weight,  Balance,  Space 

5 

8 

Ease  of  Retrofit 

5 

9 

Support  Equipment  Footprint 

10 

10 

Training  Requirements 

7 

B 

Multiple  Aircraft  MDSs 

8 

12 

Impact  on  System  Performance 

3 

13 

Impact  on  Pilot  Workload 

4 

14 

Risk  -  Risk  Factor 

7 

The  “Relative  Importance”  scores  in  Table  3.2  were  obtained  as  follows.  A  QFD 
correlation  matrix  was  set  up,  as  shown  in  Table  3.3.  Rows  correspond  to  customer 
requirements,  and  columns  correspond  to  evaluation  criteria.  Rankings  of  L  (Low),  M 
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(Medium),  H  (High),  or  blank  were  assigned  to  each  cell,  based  on  the  relative  importance  of  an 
evaluation  criterion  to  a  customer  requirement.  Numerical  scores  for  the  rankings  were  L  =  2,  M 
=  5,  H  =  8.  A  “Total  Score”  was  computed  for  each  evaluation  criterion  by  multiplying  the 
ranking  score  for  each  cell  with  the  corresponding  customer  priority  number,  then  summing 
these  products  for  each  column.  The  “Relative  Importance”  of  each  evaluation  criterion  was 
obtained  by  normalizing  the  total  score  values  to  a  maximum  of  10. 


Table  3.3.  Quality  Function  Deployment  (QFD)  Correlation  Matrix  to  Correlate 
Evaluation  Criteria  with  Customer  Requirements 


Customer  Requirements 

Customer  Priority 

_ COST _ 1 

nwaatl 

TECHNOLOGY  IMPACT  1 

1 

R&D  Required  to  Mature 

Initial  Acquisition 

Sustaining  Cost 

Benefit 

Time  Required  to  Mature 
Technology 

>. 

15 

TO 

1 

2 

o 

<  - 

Aircraft  Weight,  Balance,  Space 

Ease  of  Retrofit 

Support  Equipment  Footprint 

Training  Requirements 

Multiple  Aircraft  MDSs 

Impact  on  System  Performance 

Impact  on  Pilot  Workload 

Risk  Factor 

Applicable  to  multiple  Legacy 
systems 

5 

M 

L 

H 

L 

L 

M 

H 

L 

Improve  fault  isolation 
Derfromance 

5 

L 

H 

L 

■ 

L 

5 

H 

H 

M 

M 

L 

Minimal  impact  on  existing 
system  desian 

5 

L 

M 

L 

H 

H 

L 

H 

H 

L 

Reduce  loaistics  footDrint 

4 

L 

H 

IS 

L 

AEF  deDlovable 

4 

L 

L 

M 

M 

■H 

L 

Minimal  trainina  reauirement 

3 

— 

_ L  .  ... 

_ L. 

L 

H 

L 

L 

L 

DeDlovable  bv  2005 

3 

KH 

L 

H 

L 

M 

■ 

Hm 

L 

Commonality  of  Parts  and 
EauiDment 

3 

L 

L 

L 

L 

M 

L 

H 

L 

Affordable 

2 

H 

H 

H 

M 

M 

L 

Low  Risk 

1 

H  - 

Totals 

mm 

mm 

mm 

20 

23 

mm 

mm 

mm 

wrm 

mm 

8 

10 

mm 

Relative  Importance 

L_z_ 

8 

6 

MM 

1  10 

ZJZ 

L 

3 

4 

ZzJ 

3.1.3.  K-T  Scoring  Method 

The  K-T  scoring  procedure  involved  an  application  of  the  evaluation  criteria  and  relative 
importance  numbers,  discussed  in  the  preceding  section,  along  with  input  data  from  several 
different  sources.  The  determination  of  numerical  scores  for  the  K-T  analysis  is  discussed  first. 


An  example  K-T  scoring  matrix  is  shown  in  Table  3.4.  The  five  columns  include  the 
evaluation  criteria  (Section  3.3.2),  weight  (equal  to  Relative  Importance  from  Table  3.3.), 
rationale  for  the  score,  the  score  on  a  scale  of  0  to  10,  and  the  product  weight  x  score.  Some 
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evaluation  criteria  list  positive  characteristics,  whereas  some  list  negative  characteristics;  the 
score  is  assigned  using  the  convention  that  10  implies  the  best  possible  outcome  and  0  implies 
the  worst  possible  outcome.  The  total  weighted  score  is  the  sum  of  the  weight  x  score  products. 
The  normalized  total  weighted  score  is  the  percentage  of  the  maximum  possible  score  obtained. 
The  maximum  possible  score  of  900  assumes  a  perfect  score  of  10  for  each  evaluation  criterion. 
In  the  example  shown  in  Table  3.4,  a  normalized  score  of  49.6%  was  obtained. 

The  scores  assigned  in  the  K-T  scoring  matrix  were  obtained  from  three  different  sources. 
Criteria  in  the  “Cost”  category  were  scored  based  on  the  results  of  the  cost/benefit  analysis, 
discussed  in  Section  3.1.4.  Criteria  in  the  “Schedule”  and  “Technology  Impact”  categories  were 
scored  based  on  vendor  questionnaire  responses,  with  screening  and  modification  by  the  project 
team  based  on  the  study  of  technology  information  and  lessons  learned  during  the  interviews 
with  maintenance  personnel.  The  score  for  the  “Risk  Factor”  category  was  obtained  as  a  result 
of  the  risk  analysis  discussed  in  Section  3.1.5. 

3.1.4.  Cost/Benefit  Analysis 


3. 1.4.1.  Definitions 


The  cost/benefit  analysis  was  performed  to  obtain  estimates  of  monetary  costs  and 
benefits  associated  with  maturing  and  deploying  an  AD  technology.  Cost  was  divided  into  three 
subcategories: 


1 .  R&D  costs 

2.  Acquisition  costs 

3.  Sustaining  costs 


Both  the  sustaining  costs  and  benefits  were  computed  assuming  an  average  remaining 
service  life  of  20  years  for  the  current  fleet  of  USAF  Legacy  aircraft.  The  following  general 
formulas  were  used  to  compute  the  various  cost  and  benefit  numbers: 
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R&D  Costs  -  One-time  development  costs  to  mature  an  AD  technology  and  demonstrate 
its  implementation  on  one  aircraft  MDS. 

Acquisition  Costs  -  One-time  cost  to  acquire  an  AD  system.  This  includes 
implementation  on  multiple  MDSs,  all  MDS-specific  design  and  development  work,  purchase  of 
all  equipment  and  software,  installation  of  on-board  systems,  etc.  Because  some  AD 
technologies  include  on-board  systems  or  hardware,  the  calculation  of  acquisition  costs  was 
based  on  the  projected  cost  per  aircraft.  The  total  acquisition  cost  was  then  given  by  the 
following  equation: 

(Acquisition  Cost)  =  (Acquisition  cost  per  AC)x(Number  of AC). 

Sustaining  Costs  -  This  cost  category  accounts  for  the  cost  of  maintaining  the  on-board 
and/or  ground-based  equipment  associated  with  an  AD  technology,  including  the  cost  of  spares. 
Because  some  AD  technologies  include  on-board  systems  or  hardware,  the  calculation  of 
sustaining  costs  was  based  on  the  projected  cost  per  aircraft  per  year.  The  total  20  year  life-cycle 
sustaining  cost  was  then  given  by  the  following  equation: 


(Total  Sustaining  Cost)  =  (Sustaining  cost  per  AC  per  year)  x(Number  of AC)x(20  years). 

Benefit  -  The  anticipated  monetary  benefits  of  a  deployed  AD  system  were  provided  by 
technology  vendors  as  a  part  of  the  survey  questionnaires  (see  Section  3.3).  However,  it  was 
apparent  from  the  responses  that  in  some  cases,  different  responses  were  based  on  different 
assumptions,  and  in  many  cases,  projected  benefits  were  completely  speculative,  with  no 
quantitative  basis.  Therefore,  a  method  was  developed  to  estimate  potential  benefits  using  a 
common  basis  for  all  technologies. 
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Table  3.4.  Example  K-T  Scoring  Matrix 


Evaluation  Criterion 

Wt. 

Rationale 

Score 

(0-10) 

Wt.  x 

Score 

COST 

_ _J 

R&D  Required  to 
Mature 

7 

Comments  of  the  grader  relating  to  the  Score  assignment  are  listed  here. 

5 

35 

Initial  Acquisition 

6 

Mm 

24 

Sustaining  Cost 

2 

6 

12 

Benefit 

8 

7 

56 

SCHEDULE 

_ 

Time  Required  to 
Mature 

6 

9 

54  1 

TECHNOLOGY  IMP 

ACT  1 

Aircraft  Availability 

10 

0 

0 

Aircraft  Weight, 
Balance,  Space 

5 

2 

10 

Ease  of  Retrofit 

5 

15 

Support  Equipment 
Footprint 

10 

H 

40 

7 

5 

35 

8 

6 

48 

Impact  on  System 
Performance 

3 

7 

21 

Impact  on  Pilot 
Workload 

4 

8 

32 

Irisk 

[Risk  Factor 

[7 

9 

63 

Total  Weighted  Score  445 


Normalized  Total  Weighted  Score  49.4% 


The  first  step  was  to  estimate  a  baseline  benefit  value  representing  the  maximum  possible 
savings  attainable  with  a  100%  effective  AD  system.  The  next  step  was  to  estimate  the  fraction 
of  this  benefit  that  each  AD  system  concept  could  be  expected  to  provide  based  on  the  features 
of  the  system,  and  the  judgement  of  the  project  team  of  the  value  of  each  system  feature.  This 
judgement  was  by  necessity  somewhat  subjective,  but  all  AD  system  concepts  were  judged  on  an 
equal  basis. 

3. 1.4.2.  Calculations 


As  a  part  of  the  survey  questionnaire  (Section  3.3),  AD  technology  vendors  were  asked  to 
provide  their  cost  estimates  in  the  three  categories  (R&D  Required  to  Mature  Technology, 
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Acquisition  Cost  per  Aircraft,  and  Sustaining  Cost  per  Aircraft  per  Year).  The  vendor-supplied 
values  were  used  to  establish  characteristic  values  for  the  three  categories.  To  determine  specific 
cost  values  (and  scores  for  the  K-T  analysis)  the  vendor-supplied  values  were  compared  on  the 
basis  of  similar  technological  approaches,  and  where  obvious  discrepancies  were  found,  the 
values  were  modified  for  use  in  the  calculations. 

Example  Cost  Calculations:  Assume  that  an  example  technology  features  the  following 
unit  cost  values: 


•  R&D  required  to  mature  =  $  1 .75M 

•  Initial  acquisition  per  AC  =  S5K  per  AC 

•  Maintenance  per  year  per  AC  =  $  1 .75K  per  AC  per  Y ear 


Consistent  with  the  baseline  benefit  calculation  performed  in  Section  2.4,  the  fleet  size 
basis  corresponded  to  F-16C  Block  40/50  and  F-16D  Block  40/50  for  a  total  of  638  aircraft.  A 
mean  remaining  AC  life  of  20  years  was  used.  The  following  cost  numbers  were  obtained: 


•  Initial  acquisition  =  ($5000/AC)x(638  AC)=$3. 19M 

•  Maintenance  (sustainment)  per  year  =  ($l,750/AC/yr)x(638  AC)  =  $1.12M/yr 

•  20 Year  Life-Cycle  Cost  =  $  1 .75M  +  $3 . 1 9M  +  $1 . 1 2M/yr  x  (20  yr)  =  S27.3M 


3.1.5.  Risk  Analysis 


A  risk  analysis  was  performed  to  establish  the  score  (1-10)  for  the  risk  evaluation 
criterion  in  the  K-T  scoring  matrix.  The  purpose  of  the  risk  analysis  was  to  account  for 
uncertainties  associated  with  proposed  technologies  that  threaten  the  goals  of  technology 
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application,  or  threaten  to  compromise  critical  characteristics  and  performance  of  the  current 
system.  A  risk  analysis  matrix  was  created  for  each  AD  technology  to  determine  a  risk  score. 
The  risk  analysis  matrix  was  based  on  a  list  of  risk  elements  that  was  formed  by  the  project  team 
and  approved  by  the  government  technical  monitor.  A  total  of  12  risk  elements  were  identified 
in  the  following  four  categories: 


1.  Technical  risk 

2.  Financial  risk 

3.  Schedule  risk 

4.  Operational  risk 


These  risk  elements  are  listed  in  the  example  Risk  Assessment  matrix  shown  in  Table  3.5. 

The  scoring  for  each  risk  element  involved  two  grades  that  are  reflected  in  the  two 
grading  columns  in  Table  3.5.  Grades  of  L(ow),  M(edium)  and  H(igh)  were  assigned,  with  the 
following  numerical  equivalencies:  L=2,  M=5,  and  H=8.  The  first  grade,  “Impact  of 
Occurrence,”  was  a  technology-independent  grading  of  the  severity  of  the  impact  should  a  risk 
element  occur.  The  impact  of  occurrence  grades,  shown  in  Table  3.5,  were  determined  by  the 
project  team  and  approved  by  the  government  technical  monitor.  A  “High”  impact  of  occurrence 
corresponds  to  an  undesirable  outcome.  The  second  grade,  “Likelihood  of  Occurrence,”  was 
technology-specific.  Example  dummy  input  values  for  likelihood  of  occurrence  are  shown  in 
Table  3.5.  The  number  in  the  “Score”  column  for  each  risk  element  is  the  product  of  the 
numerical  scores  for  impact  of  occurrence  and  likelihood  of  occurrence.  The  “Total  Score”  at 
the  bottom  of  the  table  is  the  sum  of  the  all  risk  element  scores.  The  range  of  possible  total 
score  values  was  126  to  504  for  minimum  to  maximum  risk.  The  K-T  input  value  was  obtained 
by  a  linear  mapping  of  the  total  score  range  126  to  504  to  the  range  10  to  0  (Least  risk:  126  maps 
to  10;  Most  risk:  504  maps  to  0). 
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Table  3.5.  Example  Risk  Assessment  Matrix 


ID 

No. 

Risk  Element 

Oc 

ImcurL/ 

LiOc 
kl*  cur  , 

of  H) 

Sco 

re 

TECHNICAL  RISK 

1 

Redesign  of  existing  system  required 

H 

L 

16 

2 

Technology  concept  not  a  comprehensive  AD  system 

H 

L 

16 

3 

Required  manufacturing  processes  not  available 

M 

L 

10 

D 

High-uncertainty  developments  required 

L 

L 

4 

5 

Technology  not  tested 

L 

M 

10 

FINANCIAL  RISK 

6 

R&D  plan  not  well  defined 

M 

M 

25 

7 

Cost  uncertainty  for  technology  insertion 

M 

H 

40 

SCHEDULE  RISK 

8 

R&D  plan  not  well  defined 

M 

M 

25 

9 

Key  technologies  not  demonstrated 

L 

L 

4 

OPERATIONAL  RISK 

10 

High  level  of  user  training 

H 

L 

16 

11 

Increased  fault  isolation  time 

H 

M 

40 

12 

Maintainability  problems 

M 

M 

25 

\m 

Total  Score 

231 

■ 

7 

1 

Scale:  126<->504  maps  to  10<->0  for  K-T  input  (reverse  scale) 

3.2.  IDENTIFICATION  AND  SELECTION  OF  TECHNOLOGIES 


A  search  was  performed  to  identify  companies  and  organizations  actively  pursuing 
research  and  development  in  the  field  of  advanced  diagnostics  for  aircraft.  Searches  were 
performed  using  the  following  resources: 


•  Documentation  (papers,  conference  reports,  etc.)  supplied  by  AFRL/HESS. 

•  Internet  searches. 

•  Literature  and  referrals  obtained  during  interviews  with  USAF  maintenance  personnel. 
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Referrals  from  technology  vendors. 


After  compiling  information  on  the  various  technologies,  each  was  subjected  to  a 
screening  process.  The  features  and  attributes  of  each  technology  were  evaluated  against  the 
goals  of  the  project.  Technologies  were  selected  that  could  be  expected  to  contribute  to  a 
reduction  in  CNDs,  RTOKs,  etc.  and/or  improve  aircraft  readiness  by  decreasing  diagnostics  and 
maintenance  time;  or  could  be  retrofitted  onto  existing  aircraft  platforms. 

Upon  completion  of  the  screening  process,  twenty  candidate  technologies  were  identified 
for  the  formal  analysis  process.  For  each  technology,  the  formal  analysis  included: 

•  A  more  in-depth  analysis  of  the  technology  attributes. 

•  Detailed  study  of  technical  and  product  literature. 

•  Evaluation  of  questionnaire  response  (if  received). 

•  Field  reports  and  testimonials,  if  available. 

•  K-T  analysis  (including  risk  analysis  and  cost  /  benefit  analysis). 


During  the  execution  of  the  detailed  analysis,  six  of  the  twenty  AD  system  concepts  were 
rejected  for  K-T  analysis  for  one  of  the  following  reasons: 


•  The  system  concept  was  only  a  technology  component,  rather  than  an  AD  solution. 

•  The  system  was  not  defined  in  sufficient  detail  to  allow  assessment. 


Fourteen  AD  system  concepts  were  therefore  subjected  to  the  full  K-T  analysis. 
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3.3.  TECHNOLOGY  SURVEY  QUESTIONNAIRE 


A  survey  questionnaire  was  sent  to  commercial  and  government  organizations  associated 
with  approximately  twenty  AD  technology  concepts  (see  Section  3.2).  The  purpose  of  the 
questionnaire  was  to  solicit  a  consistent  body  of  knowledge  about  each  technology  that  would 
facilitate  the  comparison  of  features,  costs,  and  benefits  for  the  various  concepts  using  the  K-T 
analysis  method  described  in  Section  3.1 .  The  questionnaire  requested  the  following 
information: 


1.  A  narrative  vision  for  a  complete  AD  system. 

2.  A  narrative  description  of  the  current  status  of  required  technologies. 

3 .  A  narrative  description  of  a  technology  maturation  program. 

4.  Estimates  for  K-T  analysis  inputs  in  the  areas  of  cost/benefit  and  scheduling  (see 
subsections  3.1.3,  and  3.1.4). 

Self-assessment  on  all  K-T  analysis  evaluation  criteria  related  to  technology  impact  and  risk 
(see  subsections  3.1.3,  3.1.5). 


SECTION  4.  DISCUSSION  OF  RESULTS 


This  section  contains  a  general  discussion  of  the  results  of  the  AD  technology  survey  and 
the  K-T  analysis.  The  technology  survey  incorporated  information  from  two  sources:  the 
general  literature  on  various  and  proposed  technologies  related  to  AD,  and  the  AD  system  vision 
statements  and  self-assessments  provided  by  technology  vendors  in  response  to  the  technology 
questionnaire  discussed  in  Section  3.3. 

The  AD  system  concepts  generally  drew  from  a  finite  set  of  technology  components, 
explained  in  Section  4.1.  A  major  theme  developed  during  the  study  was  the  need  to 
differentiate  between  diagnosis  and  maintenance  of  the  avionics  “infrastructure”  (communication 
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busses  and  permanent  wiring),  and  diagnosis  of  LRU  faults.  This  is  further  discussed  in  Section 
4.2.  Results  of  the  K-T  analysis  are  discussed  in  Section  4.3. 


4.1.  AD  TECHNOLOGY  COMPONENTS 


Each  technology  reviewed  featured  one  or  more  of  the  technology  components  described 
in  the  following  paragraphs. 

Communication  Bus  and  Wiring  Maintenance'.  The  communication  busses  and 
permanent  wiring  found  on  an  aircraft  represent  test  voids;  i.e.  the  systems  are  not  subjected  to 
routine  maintenance.  Wiring  and  busses  degrade  over  time,  causing  intermittent  failures  during 
flight  or,  in  the  case  of  degraded  busses,  sluggish  response  of  aircraft  systems.  Field 
implementations  have  shown  that  performing  routine  maintenance  on  the  bus  systems  can 
significantly  reduce  diagnostic  problems  and  improve  aircraft  availability. 

In-Flight  Data  Capture:  CNDs  reflect  an  inability  to  duplicate  on  ground  a  fault 
indication  detected  during  flight.  This  can  result  from  environmental  conditions  (temperature, 
moisture,  vibration,  etc.)  or  aircraft  system  states  associated  with  flight  that  are  not  duplicated 
during  ground  tests.  Therefore,  many  maintenance  personnel  believe  that  diagnostic 
improvements  can  be  achieved  by  obtaining  better  documentation  of  system  information  at  the 
time  a  fault  indication  occurs.  Most  aircraft  already  have  some  form  of  on-board  data  capture, 
but  this  is  usually  limited  to  highly  filtered  fault  indications  and/or  basic  flight  condition  data. 
More  comprehensive  on-board  in-flight  data  capturing  systems  are  proposed.  The  additional 
data  could  be  used  for  real-time  in-flight  diagnostics,  or  for  post-flight,  ground-based 
diagnostics. 

On-Board  Diagnostics:  An  on-board  diagnostic  system  would  continuously  monitor  for 
fault  indications,  and  would  process  fault-related  information  in  real  time  to  identify  a  faulty 
component.  While  newer  platform  designs  such  as  the  F-22  and  JSF  have  plans  for  advanced 
on-board  diagnostic  systems,  such  a  system  has  not  been  implemented  on  a  Legacy  aircraft 
outside  of  BITs. 
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Distributed  Diagnostics:  A  distributed  diagnostic  system  generally  includes  multiple 
distributed  components  (reflecting  subsystem  monitoring)  throughout  an  aircraft,  plus  a  central 
diagnostic  component  to  integrate  the  results  and  provide  higher  level  diagnostics.  This  general 
concept  may  be  impractical  for  Legacy  aircraft  due  to  the  high  retrofit  costs. 

Advanced  Diagnostic  Algorithms:  There  is  a  large  family  of  proposed  methods  for 
performing  the  analytical  diagnostic  process.  Many  of  these  methods  incorporate  some  form  of 
artificial  intelligence  (e.g.,  neural  networks,  fuzzy  logic)  and  machine  learning.  Some  work 
focuses  on  efficient  methods  for  building  system  models.  It  is  proposed  that  information  in 
maintenance  history  databases  be  processed  to  contribute  to  improved  diagnostic  rules. 

Interactive  Diagnostics:  An  AD  system  that  includes  interactive  software  has  the 
capability  of  leading  a  technician  step-by-step  through  the  entire  diagnostic  and  repair  process. 
This  feature  can  decrease  training  requirements  while  maintaining  a  system’s  performance  level. 

Integrated  Maintenance  Systems:  This  title  represents  a  comprehensive  maintenance  and 
diagnostic  system  under  which  all  aspects  of  maintenance  and  diagnostics  are  coordinated, 
including  maintenance  data,  inventory,  scheduling,  repair  time,  etc. 

Air-to-Ground  Data  Transfer:  Status  and/or  fault  data  that  is  automatically  transmitted  to 
a  ground-based  station  offers  the  ability  to  perform  ground-based  diagnostics  while  an  aircraft  is 
in  flight.  This  could  enable  maintenance  personnel  to  prepare  for  repair  actions  before  the 
aircraft  arrives,  thereby  reducing  turnaround  time  and  increasing  sortie  rate. 


4.2.  DIAGNOSIS  AND  MAINTENANCE  OF  COMMUNICATION  BUSSES  AND 
PERMANENT  WIRING 


A  strong  theme  that  emerged  during  the  study  was  that  communication  busses  and 
permanent  wiring  fell  into  a  test  void;  i.e.  they  were  not  subjected  to  regular  diagnosis  and 
maintenance.  The  busses  and  wiring  represent  the  “infrastructure”  on  which  avionics  systems 
are  based.  Inherent  in  LRU  BITs  and  general  fault  isolation  procedures  is  the  assumption  that 
busses  and  wiring  are  performing  properly.  The  health  of  permanent  wiring  in  aging  commercial 
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aircraft  has  become  a  well-publicized  maintenance  concern  in  recent  years,  and  aging  Air  Force 
aircraft  face  the  same  concern. 

There  are  current  USAF  programs  underway  to  review  maintenance  practices  and 
identify  diagnostic  tools  for  permanent  wiring,  so  this  issue  will  not  be  considered  further  in  this 
report.  However,  there  is  evidence  that  neglected  bus  maintenance  may  constitute  an  even  larger 
problem  to  avionics  diagnostics  than  faulty  wiring.  The  “Proud  Falcon”  program  was  started  at 
Nellis  Air  Force  Base  in  the  early  1990s  based  on  the  premise  that  that  a  significant  number  of 
CNDs,  RTOKs,  and  in-flight  intermittences  were  the  result  of  degraded  or  out-of-spec  bus 
systems,  rather  than  failed  LRUs.  The  Multiplex  Bus  Fault  Isolator  (MBFI)  was  developed  in 
response.  Most  LRUs  on  the  F-16  communicate  via  MIL-STD  1553  communication  busses. 
Using  the  MBFI,  the  Proud  Falcon  crew  performed  maintenance  on  the  busses  and  routinely 
found  problems  such  as: 


•  Faulty  connectors  (damaged,  worn,  loose,  dirty) 

•  Unshielded  cabling 

•  Faulty  splices 

•  Out-of-spec  stub  lengths 

•  Use  of  incorrect  transformers 

•  Out-of-tolerance  couplers 

•  Shorts  to  ground 

•  Out-of-spec  transformers 


Proud  Falcon  first  used  the  MBFI  to  inspect  the  Enhanced  Central  Interface  Unit  (ECIU) 
busses.  After  12  months,  the  average  ECIU  failure  rate  fell  from  16.25  to  4.3  units  per  quarter. 
The  estimated  saving  was  $1,186,157.  A  1995  study  compared  the  CND  rates  of  these  aircraft 
with  the  rest  of  the  Air  Force  F-16  Block  40s.  TICARS  was  used  to  collect  data  for  five  LRUs. 
The  average  CND  rate  for  all  five  LRUs  was  32%  for  the  Air  Force  overall,  while  the  Proud 
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Falcon  CND  rate  was  4.4%.  It  was  estimated  that  if  the  Air  Force  used  the  MBFI  to  clean  up  the 
busses  on  Block  40  aircraft  alone,  it  would  realize  a  unit  cost  savings  of  $38,498,606  and 
exchange  cost  savings  of  $4,027,336  each  year. 

The  MBFI  is  currently  supported  by  TISMD  at  Hill  AFB,  and  is  deployed  as  standard 
equipment  at  all  F-16  bases.  User  feedback  attests  to  the  value  of  the  MBFI  at  reducing  CNDs 
and  improved  aircraft  availability.  However,  the  MBFI  requires  a  high  level  of  user  training  and 
experience,  and  this  has  prevented  it  from  being  adopted  as  a  routine  maintenance  tool  at  most  F- 
16  bases.  Commercial  bus  diagnostic  tools  and  software  that  expand  on  the  capabilities  of  the 
MBFI  have  been  produced. 

Why  has  bus  maintenance  not  been  more  widely  recognized  as  a  key  factor  in 
diagnostics?  The  explanation  seems  to  be  that  the  MIL-STD  1553  bus  is  extremely  robust,  and 
continues  to  perform  its  basic  communications  functions  even  in  a  highly  degraded  state,  albeit 
with  greatly  reduced  efficiency. 

Based  on  the  indications  of  the  large  influence  of  bus  maintenance  on  diagnostic 
effectiveness,  it  is  concluded  that  busses  must  be  maintained  as  the  starting  point  for  the 
deployment  of  an  AD  system.  This  could  be  implemented  either  through  routine  bus 
maintenance  performed  using  tools  such  as  the  MBFI,  or  by  incorporating  bus  health  monitoring 
and  diagnostics  as  part  of  an  AD  system. 

Since  the  MBFI  bus  diagnostic  tool  is  already  standard  equipment  at  F-16  bases,  the 
additional  costs  for  complete  implementation  of  the  MBFI  would  be  small,  consisting  of  the 
cost  of  additional  technician  training,  and  the  labor  costs  during  the  bus  maintenance  operations. 
Based  on  limited  studies  with  the  MBFI  and  related  commercial  products,  the  benefits  of  routine 
bus  maintenance  in  terms  of  reduced  CNDs  and  other  improvements  to  diagnostic  efficiency 
may  be  very  large. 
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4.3.  ADVANCED  DIAGNOSTIC  CONCEPTS 


4.3.1.  General  Comments 


The  survey  of  AD  technologies  incorporated  information  from  two  sources.  The  first 
source  was  general  technical  and  product  literature  on  various  and  proposed  AD  technologies. 
The  second  source  was  the  survey  questionnaire  responses  provided  by  technology  vendors.  The 
survey  questionnaire  responses  were  designed  to  solicit  both  a  complete  AD  system  vision  from 
the  vendor  and  a  self-assessment  of  the  system  concept  with  respect  to  the  evaluation  criteria 
(including  risk,  cost,  and  benefit)  used  in  the  K-T  Analysis.  It  is  recognized  that  the  vendors 
were  only  able  to  put  minimal  effort  into  the  voluntary  questionnaire,  and  that  their  responses  are 
therefore  preliminary  and  cursory.  Therefore,  the  K-T  analysis  inputs  were  made  with 
consideration  of  the  vendor’s  self-assessment,  but  also  using  the  judgement  of  the  project  team 
reflecting  the  literature  information,  the  lessons  learned  from  the  interviews  (Section  2.1),  and 
comparisons  of  the  different  AD  system  concepts. 

The  K-T  analysis  method  was  applied  to  fourteen  specific  AD  concept  proposals 
submitted  by  commercial  and  government  organizations.  However,  the  concepts  considered  do 
not  encompass  all  possible  approaches,  nor  do  they  reflect  the  vision  of  all  organizations 
involved  in  applicable  technologies.  Furthermore,  some  organizations  that  submitted 
questionnaires  have  other  AD  system  concepts  that  were  not  divulged  due  to  business  sensitive 
issues.  Recognizing  these  limitations  of  scope,  the  K-T  analysis  results  are  discussed  here  in 
terms  of  general  approaches  rather  than  specific  technology  sources. 

A  general  assessment  of  the  AD  system  concepts  is  offered.  There  was  generally  a  large 
disparity  between  the  AD  system  visions  offered  by  vendors  and  the  current  state  of 
technologies.  The  AD  concepts  were  generally  not  well  defined,  either  with  respect  to  technical 
details,  or  with  respect  to  the  development  program  required  to  mature  and  verify  a  system. 
Most  concepts  would  require  simultaneous  developments  in  multiple  disciplines,  and  involve 
complex  details  in  terms  of  hardware  design  and  fabrication,  software  development,  system 
integration,  and  incorporation  of  aircraft  configuration  information.  Because  of  the  diverse 
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nature  of  Legacy  aircraft,  the  complexity  of  any  individual  aircraft,  the  limited  research  funds 
available,  and  the  prohibitive  cost  of  major  modifications  to  existing  aircraft  systems,  the  authors 
believe  that  a  viable  AD  solution  must  be  relatively  simple  in  architecture  while  targeting  a  high- 
benefit  diagnostic  niche.  Recent  experience  has  shown  that  an  AD  concept  that  proposes  major 
revisions  to  the  current  maintenance  environment  will  be  expensive,  and  may  not  be  readily 
accepted  by  end  users.  Finally,  although  AD  concept  rankings  are  provided  in  the  following 
section,  no  single  AD  path  proved  to  be  inherently  superior  to  all  others.  Any  future  proposals 
for  AD  systems  should  be  judged  on  their  individual  merits,  with  strong  consideration  given  to 
the  degree  to  which  all  system  features  are  defined,  sources  for  all  technologies  are  identified, 
technology  maturation  steps  are  specified,  and  a  realistic  scope  is  targeted. 


4.3.2.  K-T  Analysis  Results 


Twenty  AD  system  concepts  proposed  by  14  different  commercial  and  government 
organizations  were  considered  for  K-T  analysis.  Six  concepts  were  rejected  for  final  K-T 
analysis  for  reasons  discussed  in  Section  3.2.  Four  of  the  concepts  were  related  to  diagnosis  and 
repair  of  communication  busses  and  permanent  wiring  discussed  in  Section  4.2.  The  remaining 
10  concepts  were  grouped  into  four  general  categories  for  discussion  in  subsequent  sections. 
These  four  categories  correspond  to  four  of  the  seven  technology  components  described  in 
Section  4. 1  that  reflect  the  dominant  feature  of  each  concept.  The  categories  are: 


1 .  Interactive  Diagnostic  Systems 

2.  On-Board  Diagnostic  Systems 

3.  In-Flight  Data  Capture  Systems 

4.  Integrated  Maintenance  Systems 

K-T  scoring  trends  are  based  on  these  four  categories,  and  selected  technology  features  are  noted 
and  discussed. 
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43.2.1.  Interactive  Diagnostic  Systems 


The  three  highest-scoring  technologies  involved  interactive  ground-based  diagnostics. 
This  attribute  offered  the  general  benefit  of  more  efficient  diagnostic  procedures,  and  addressed 
the  priority  of  a  user-friendly  system  that  avoids  the  need  for  highly  trained  or  specialized 
technicians. 

Two  of  the  concepts  featured  interfaces  to  communications  busses.  This  is  attractive 
from  the  standpoint  of  requiring  very  minor  modifications  to  the  aircraft,  while  tapping  into  a 
source  of  extensive  information  on  avionics  systems  performance.  One  concept  emphasized  an 
advanced  diagnostic  process  that  combines  model-based  and  case-based  reasoning  and 
incorporates  an  integrated  knowledge  base.  This  diagnostic  process  provides  the  basis  for 
natural  evolution  into  predictive  maintenance.  The  concept  also  emphasizes  the  use  of  COTS 
hardware  and  software  that  can  be  applied  to  any  aircraft,  which  is  attractive  for  the  goal  of 
serving  multiple  Legacy  aircraft  designs.  Another  feature  emphasized  in  this  category  is 
advanced  PMAs. 

The  biggest  criticism  of  interactive  (ground-based)  diagnostic  systems  is  that,  when  used 
as  the  sole  AD  system  component,  they  do  not  address  the  concern  that  a  significant  number  of 
intermittent  fault  indications  surround  the  physical  flight  environment  or  occur  because  of 
system  configurations  or  inputs  that  only  occur  in  flight. 

Using  the  vendor-supplied  unit  cost  estimates  applied  to  638  aircraft  (F-16C/D  Block  40/50) 
with  an  20-year  average  remaining  service  life,  the  predicted  life  cycle  cost  in  this  category  was 
in  the  range  $5M-$20M.  AD  solutions  in  this  category  were  predicted  to  correct  40%  to  60%  of 
diagnostic  problems  remaining  after  full  implementation  of  communications  bus  maintenance 
(which  was  estimated  to  provide  a  50%  improvement). 
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¥.3.2.2.  On=Boardm Diagnostic  Systems 


The  second  highest  scoring  group  of  technologies  was  in  the  category  of  on-board,  real¬ 
time  diagnostic  systems.  Three  AD  system  concepts  fell  into  this  category.  Real-time,  on-board 
diagnostics  can  potentially  eliminate  the  problem  of  trying  to  duplicate  a  fault  on  the  ground  that 
so  often  results  in  a  NFF  or  CND.  All  three  concepts  use  a  distributed  architecture  (see  Section 
4.1).  One  concept  emphasized  multi-level  diagnostics,  featuring  both  distributed  subsystem 
diagnostics  and  a  global  integrator  performing  higher-level  diagnostics.  This  concept  also 
emphasized  artificial  intelligence  and  mining  of  maintenance  database  information  to  optimize 
the  diagnostic  analysis.  The  second  concept  would  involve  monitoring  of  the  processor 
performance  and  software  execution  in  each  LRU.  The  third  concept  emphasized  advanced 
techniques  of  model  input  for  model-based  diagnostics. 

The  biggest  drawback  of  the  three  concepts  considered  was  the  extensive  level  of 
modifications  or  retrofitting  that  would  be  required.  All  three  concepts  seemed  to  offer 
comprehensive  diagnostic  solutions  that  would  be  practical  if  designed  in  from  the  start,  but 
which  would  require  expensive  retrofits  for  Legacy  aircraft. 

Using  the  vendor-supplied  unit  cost  estimates  applied  to  638  aircraft  (F-16C/D  Block  40/50) 
with  a  20-year  average  remaining  service  life,  the  predicted  life  cycle  cost  in  this  category  was 
in  the  range  $18M-$48M.  AD  solutions  in  this  category  were  predicted  to  correct  70%  to  80% 
of  diagnostic  problems  remaining  after  full  implementation  of  communications  bus  maintenance 
(which  was  estimated  to  provide  a  50%  improvement). 


¥.3.2.3.  Flight  Data  Capture. 


This  solution  category  may  seem  ill  chosen  in  that  it  refers  only  to  a  single  technology 
component  within  an  AD  system.  However,  it  was  singled  out  because  one  vendor  possesses  a 
demonstrated  system  for  high-speed,  in-flight  data  capture  that  could  serve  as  the  basis  for 
diagnostics  that  incorporate  the  actual  system  data  associated  with  a  fault  indication.  The 
recording  system  is  physically  compact  and  light,  and  in  its  simplest  form  requires  only  easy-to- 
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implement  interfaces  with  communications  busses.  It  features  removable,  high-volume  solid- 
state  data  storage.  It  is  easily  adaptable  to  different  aircraft  designs  by  importing  the  system 
ICD.  On  ground,  the  system  interface  incorporates  a  user-friendly  GUI.  The  proposed 
diagnostics  would  be  ground-based,  though  the  data  collection  system  opens  up  the  possibility 
for  in-flight  diagnostics.  Proposed  diagnostic  capabilities  also  include  the  playback  of  full 
system  data  for  duplicating  in-flight  inputs  to  LRUs. 

The  AD  concept  in  this  category  was  found  deficient  in  that  the  complete  AD  system 
concept  was  not  well  defined.  The  current  data-gathering  tool  requires  an  expert-level  operator. 
Little  indication  was  given  for  a  vision  of  the  system  operating  in  the  current  O-level 
maintenance  environment.  Therefore,  this  technology  was  perceived  as  an  AD  component, 
rather  than  a  complete  system  solution. 


43.2.4.  Integrated  Maintenance^  Systems 


Three  of  the  AD  system  concepts  analyzed  fell  into  the  category  of  integrated 
maintenance  systems.  These  concepts  go  far  beyond  improved  diagnostics,  incorporating  highly 
integrated  and  automated  inventory  planning  and  control,  maintenance  planning,  and 
documentation.  Some  diagnostic-related  functions  proposed  for  these  systems  are  listed  below: 


•  Incorporate  data  mining  and  machine  learning 

•  Address  the  CND  problem  by  improving  the  initial  problem  analysis. 

•  Address  the  RTOK  problem  by  continued  application  of  “expert  system”  and  “model  based” 
technologies  to  provide  directed  troubleshooting  assistance  throughout  the  fault  isolation 
process. 

•  Integrate  PMAs,  maintenance  information  systems,  data  collection  systems,  pilot  debrief, 
etc.  using  radios,  modems,  and  LANs. 

•  Use  air-to-ground  data  transfer  to  enable  early  ground-based  diagnostics  and  preparations 
for  maintenance  actions  while  an  aircraft  is  still  returning  from  a  mission. 
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The  proposed  solutions  in  this  category  are  attractive  from  the  standpoint  of  designing  an 
optimal  maintenance  system  that  automates  and  integrates  many  parts  of  the  logistics  system. 
However,  for  the  focus  of  the  current  study,  these  concepts  suffered  in  several  ways.  First,  they 
are  extremely  costly  solutions,  both  to  develop  and  deploy,  and  are  not  likely  to  be  achievable  in 
the  five-year  time  frame  specified.  Second,  they  would  require  major  changes  to  existing 
maintenance  information  systems  (such  as  CAMS)  and  major  changes  to  current  O-level 
operations.  These  factors  indicate  extreme  risks  for  the  proposed  solutions,  and  are  therefore 
judged  unacceptable  for  application  to  Legacy  aircraft. 


5.  CONCLUSIONS  AND  RECOMMENDATIONS 

A  study  was  conducted  to  explore  Advanced  Diagnostic  technologies  that  could  be 
applied  to  USAF  Legacy  aircraft  to  improve  the  diagnosis  of  faults  in  avionics  systems.  The  first 
task  was  to  review  current  diagnostic  practices,  issues,  technologies,  and  maintenance  statistics. 
The  second  task  was  to  identify  and  assess  candidate  AD  technologies.  Over  twenty 
technologies  were  identified  and  assessed. 

There  was  generally  a  large  disparity  between  the  AD  system  visions  offered  by  vendors 
and  the  current  state  of  AD  technologies.  For  the  most  part,  the  AD  concepts  lacked  definition, 
both  with  respect  to  technical  details  and  the  development  program  required  to  mature  and  verify 
a  system.  Concepts  generally  require  coordinated  development  and  activities  in  multiple  areas, 
including  hardware  design,  software  development,  system  integration,  and  incorporation  of 
aircraft  configuration  information.  Given  the  diversity  of  Legacy  aircraft,  the  complexity  of 
avionics  on  each  aircraft,  the  limited  research  funds  available,  and  the  prohibitive  cost  of  major 
modifications  to  existing  aircraft  systems,  the  authors  believe  that  a  viable  AD  solution  must  be 
relatively  simple  in  architecture  while  targeting  a  high-benefit  diagnostic  niche.  AD  concepts 
that  propose  major  revisions  to  the  current  maintenance  environment  were  judged  to  be 
unacceptable  due  to  high  cost,  difficulty  in  implementation,  and  probable  resistance  by  the  end 
users. 
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No  single  AD  concept  studied  was  judged  to  be  inherently  superior  to  all  others. 
Therefore,  any  proposed  AD  system  considered  for  development  by  the  US  AF  should  be 
assessed  on  its  individual  merits,  with  strong  consideration  given  to  the  degree  to  which  all 
system  features  are  defined,  sources  for  all  technologies  are  identified,  technology  maturation 
steps  are  specified,  and  a  realistic  scope  is  targeted.  The  following  recommendations  are  made 
as  guidelines  for  future  Air  Force  research  activities  for  maturing  and  verifying  candidate  AD 
technologies  for  aircraft  avionics: 


1 .  A  range  of  proposed  AD  solutions  appear  to  be  viable  for  development  and  application  to 
Legacy  aircraft  in  the  near  future.  The  following  features  are  given  the  highest  overall 
recommendation,  given  the  tradeoff  between  diagnostic  benefits  and  acceptable  development 
risk: 

•  Additional  on-board,  real-time  data  capture 

•  Off-board  diagnostic  processing 

•  Advanced  diagnostic  software: 

♦  Configurable,  platform-independent  interface  software 

♦  Advanced  diagnostic  algorithms  (e.g.,  artificial  intelligence,  advanced  reasoning, 
fuzzy  logic) 

♦  User-friendly  GUI  (interactive  diagnostics,  easy  operation,  low  requirements  for  user 
expertise) 

2.  Consider  supporting  the  maturation  of  an  AD  system  that  includes  on-board  diagnostics.  On¬ 
board  diagnostics  could  offer  many  optimal  benefits,  such  as  speed,  small  footprint  for 
support  equipment,  and  better  in-flight  feedback  on  aircraft  health  to  the  pilot.  However,  a 
viable  on-board  AD  system  must  emphasize  low  development  and  acquisition  cost;  ease  of 
retrofit;  and  low  impact  on  aircraft  weight,  balance,  and  space. 

3.  Properly  maintained  communications  busses  are  required  for  effective  avionics  diagnostics. 
The  MBFI  is  now  a  standard  bus  diagnostic  tool  at  all  F-16  bases,  yet  while  it  has  been 
demonstrated  to  be  a  powerful  tool,  it  is  not  being  fully  utilized  because  of  the  high  level  of 
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user  expertise  required  and  inadequate  training  of  technicians.  There  is  evidence  that  proper 
maintenance  of  communications  busses  provides  large  decreases  in  diagnostic  problems. 
Routine  maintenance  of  communication  busses  (and  permanent  wiring)  should  be 
considered.  The  assessment  of  new  AD  technologies  must  be  performed  on  the  basis  of 
properly  functioning  communication  busses. 
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AC 

AD 

AEF 

AFRL 

AIS 

BCS 

BIT 

CAMS 

CBR 

CFRS 

CND 

COTS 

DLA 

DO 

DTM 

ECIU 

FI 

GUI 

LRU 

MBFI 

MBR 

MDS 

MFL 

MOCC 

MTTR 

NDF 

NFF 

NRTS 

O-Level 

PMA 

QFD 

QPA 

REMIS 

ROI 

R&M 

RTOK 

SRU 

TO 

TPS 

TRS 

UDRI 

WTIC 


APPENDIX  A.  List  of  Abbreviations 

Aircraft 

Advanced  Diagnostics 
Air  Expeditionary  Force 
Air  Force  Research  Laboratory 
Avionics  Intermediate  Shop 
Bench-Checked  Serviceable  (I-Level) 

Built  In  Test 

Core  Automated  Maintenance  System 
Case-Based  Reasoning 
Computerized  Fault  Recorder  System 
Could  Not  Duplicate 
Commercial  Off-The-Shelf 
Defense  Logistics  Agency 
Delivery  Order 
Data  Transfer  Module  (F-15) 

Enhanced  Central  Interface  Unit 

Fault  Isolation 

Graphical  User  Interface 

Line  Replacement  Unit 

Multiplex-Bus  Fault  Isolator 

Model-Based  Reasoning 

Mission  Design  Series 

Maintenance  Fault  List 

Maintenance  Operations  Control  Center 

Mean  Time  To  Repair 

No  Defect  Found 

No  Fault  Found  (O-Level) 

Not  Repairable  This  Station 

Organizational  Level  (at  the  flight  organization) 

Portable  Maintenance  Aid 
Quality  Function  Deployment 
Quantity  Per  Aircraft 

Reliability  &  Maintainability  Information  System 
Return  On  Investment 
Reliability  &  Maintainability 
Re-Test  O.K.  (Depot  Level) 

Shop  Replacement  Unit;  usually  a  circuit  board  that  plugs  into 
Technical  Order 
Test  Program  Set 

Technology  for  Readiness  and  Sustainment 
University  of  Dayton  Research  Institute 
Work  Unit  Code 


APPENDIX  B.  REMIS  ALGORITHMS 

•  Cannot  Duplicate  (CND)  Action 

If  the  record's  first  position  of  the  Work  Unit  Code  (WUC)  is  not  equal  to  zero  (i.e.  not  a 
support  general  record),  and  Units  is  greater  than  zero  (i.e.,  this  is  a  completed  maintenance 
action)  and  the  Action  Taken  Code  equals  "H"  and  How  Malfunction  Code  equals  “799”or  “812” 
or  “948”,  then  augment  the  Number  of  Cannot  Duplicate  Actions. 

•  Bench  Check  Serviceable  (BCS)  Action 

Add  one  each  time  the  first  two  positions  of  the  on-equipment  WUC  is  not  equal  to  "01" 
through  "09"  and  the  How  Malfunction  Code  Classification  equals  1  or  2  and  the  Action  Taken 
Code  equals  "P"  or  "R"  and  there  is  a  related  off-equipment  record  whose  WUC  equals  this  on- 
equipment  record's  WUC  and  its  Action  Taken  Code  equals  "B"  and  its  Type  Maintenance  Code 
is  not  equal  to  "R"  (deport)  or  its  When  Discovered  Code  is  not  equal  to  "S"  (during  depot 
maintenance). 

•  Unscheduled  Removal 

If  the  first  two  positions  of  the  on-equipment  or  off-equipment  WUC  is  not  equal  to  "01" 
or  "02"  or  "03"  or  "04"  or  "05"  or  "06"  or  "07"  or  "08"  or  "09"  and  the  Type  Equipment  equals 
"A"  and  the  Type  Maintenance  Code  equals  "B"  or  "S"  or  "Y"  and  the  Action  Taken  Code  equals 
"P"  or  "R"  and  the  How  Malfunction  Code  Classification  equals  1  or  2  and  there  is  not  a  related 
off-equipment  record  whose  WUC  equals  this  on-equipment  record's  WUC  and  its  Action  Taken 
Code  equals  "B",  then  augment  Units  to  Removals  (Unscheduled). 

•  Retest  OK  (RTOK) 

Add  one  unit  each  time  the  first  two  positions  of  the  WUC  is  not  equal  to  "01"  through 
"09"  and  the  Action  Taken  Code  of  the  off  equipment  record  equals  "B"  and  its  Type 
Maintenance  Code  equals  "R"  (depot)  or  its  When  Discovered  Code  equals  "S"  (during  depot 
maintenance). 
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MTTR  (On-Equipment) 


REPAIR  HOURS  (ON) 

MTTR  - - 

REPAIR  ACTIONS  (ON) 


•  MTTR  (Off-Equipment) 

REPAIR  HOURS  (OFF) 

MTTR  = - 

REPAIR  ACTIONS  (OFF) 


Source:  AFCSM  25-524,  Volume  III,  1  August  1996 
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